I saw you introduce new example files. Is it possible to reuse script we already have?

Fix conflicts, thanks

Thanks for the review! I rebased the PR on main, resolved conflicts, and pushed the updated branch.

V2 (Wan2.1) updates:

• Added a minimal fallback in OmniDiffusion for AniSora V2/V3 repos that don’t ship model_index.json, so _class_name resolves to AniSoraV2I2VPipeline.
• Fixed CLIP conditioning in the V2 pipeline to prefer pil_image over preprocessed_image (avoids PIL conversion errors during warmup).
• For large-model runs (e.g., AniSora V2), I used PYTORCH_ALLOC_CONF=expandable_segments:True to reduce allocator fragmentation on big GPUs.

V1 (CogVideoX) updates:

• Added CogVideoXImageToVideoPipeline alias in the registry for the 5B model.
• Added pre/post-process hooks for the CogVideoX pipeline.

Shared updates (V1 + V2):

• Implemented required forward() + load_weights() for vLLM integration.
• Fixed image preprocessing & device handling and corrected self.dtype usage.
• Declared support_image_input=True and ensured model weights are moved to device before inference.

Testing:

• V1 (CogVideoX) ran on RTX 6000 S without setting PYTORCH_ALLOC_CONF.
• V2 (Wan2.1) ran on H200 (short config) with the allocator env set.

@dorhuri123 It seems that the first video has accuracy problems

@ZJY0516 agreed — the first output looks clearly off (strong color inversion / desaturation compared to the input). I re‑ran the exact same settings and got a cleaner output on my side(after all the changes that were done to suite the existing example file).

Could you point to the specific behavior you want to treat as the “accuracy issue” (e.g., color inversion, identity drift, motion artifacts)? That would help me isolate whether it’s still a code path issue or just variability.

Same input/settings for both runs:

attempt 1

attempt 2

The shape and movement of the cat in the video don’t look quite right. Could you compare this with the official implementation to verify? @dorhuri123

@ZJY0516 I compared against the official Diffusers CogVideoXImageToVideoPipeline using the same input/settings on an RTX 6000 (Blackwell Server Edition). The output shows the same shape/motion characteristics as the vLLM run, so it looks like this is model behavior rather than an integration issue.

Baseline (official diffusers) commands + script used:

# env
python -m venv ~/anisora-diffusers
source ~/anisora-diffusers/bin/activate
pip install --upgrade pip
pip install diffusers==0.36.0 transformers accelerate safetensors huggingface_hub \
            sentencepiece tiktoken protobuf imageio imageio-ffmpeg
pip install --pre --upgrade torch --index-url https://download.pytorch.org/whl/nightly/cu128

# input image
wget -O /tmp/cat.png https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/cat.png

# run_diffusers_anisora_v1.py
import torch
import PIL.Image
from diffusers import CogVideoXImageToVideoPipeline
from diffusers.utils import export_to_video

pipe = CogVideoXImageToVideoPipeline.from_pretrained(
    "Disty0/Index-anisora-5B-diffusers",
    torch_dtype=torch.bfloat16,
).to("cuda")

image = PIL.Image.open("/tmp/cat.png").convert("RGB")

video = pipe(
    prompt="A cat playing with yarn",
    image=image,
    height=480,
    width=720,
    num_frames=81,
    num_inference_steps=50,
    guidance_scale=5.0,
    output_type="np",
).frames[0]

export_to_video(video, "anisora_v1_diffusers.mp4", fps=16)

I’ll attach the diffusers output video in this comment. If you’re seeing a specific artifact you want addressed, let me know the exact behavior and I’ll dig deeper.

resolve conflicts please

@ZJY0516 I compared against the official Diffusers CogVideoXImageToVideoPipeline using the same input/settings on an RTX 6000 (Blackwell Server Edition). The output shows the same shape/motion characteristics as the vLLM run, so it looks like this is model behavior rather than an integration issue.

Baseline (official diffusers) commands + script used:

# env
python -m venv ~/anisora-diffusers
source ~/anisora-diffusers/bin/activate
pip install --upgrade pip
pip install diffusers==0.36.0 transformers accelerate safetensors huggingface_hub \
            sentencepiece tiktoken protobuf imageio imageio-ffmpeg
pip install --pre --upgrade torch --index-url https://download.pytorch.org/whl/nightly/cu128

# input image
wget -O /tmp/cat.png https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/cat.png

# run_diffusers_anisora_v1.py
import torch
import PIL.Image
from diffusers import CogVideoXImageToVideoPipeline
from diffusers.utils import export_to_video

pipe = CogVideoXImageToVideoPipeline.from_pretrained(
    "Disty0/Index-anisora-5B-diffusers",
    torch_dtype=torch.bfloat16,
).to("cuda")

image = PIL.Image.open("/tmp/cat.png").convert("RGB")

video = pipe(
    prompt="A cat playing with yarn",
    image=image,
    height=480,
    width=720,
    num_frames=81,
    num_inference_steps=50,
    guidance_scale=5.0,
    output_type="np",
).frames[0]

export_to_video(video, "anisora_v1_diffusers.mp4", fps=16)

I’ll attach the diffusers output video in this comment. If you’re seeing a specific artifact you want addressed, let me know the exact behavior and I’ll dig deeper.

anisora_v1_diffusers.mp4

which transformers version you are using?

did you keep the seed as the same for comparision with diffusers?

@ZJY0516 I compared against the official Diffusers CogVideoXImageToVideoPipeline using the same input/settings on an RTX 6000 (Blackwell Server Edition). The output shows the same shape/motion characteristics as the vLLM run, so it looks like this is model behavior rather than an integration issue.
Baseline (official diffusers) commands + script used:

# env
python -m venv ~/anisora-diffusers
source ~/anisora-diffusers/bin/activate
pip install --upgrade pip
pip install diffusers==0.36.0 transformers accelerate safetensors huggingface_hub \
            sentencepiece tiktoken protobuf imageio imageio-ffmpeg
pip install --pre --upgrade torch --index-url https://download.pytorch.org/whl/nightly/cu128

# input image
wget -O /tmp/cat.png https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/cat.png

# run_diffusers_anisora_v1.py
import torch
import PIL.Image
from diffusers import CogVideoXImageToVideoPipeline
from diffusers.utils import export_to_video

pipe = CogVideoXImageToVideoPipeline.from_pretrained(
    "Disty0/Index-anisora-5B-diffusers",
    torch_dtype=torch.bfloat16,
).to("cuda")

image = PIL.Image.open("/tmp/cat.png").convert("RGB")

video = pipe(
    prompt="A cat playing with yarn",
    image=image,
    height=480,
    width=720,
    num_frames=81,
    num_inference_steps=50,
    guidance_scale=5.0,
    output_type="np",
).frames[0]

export_to_video(video, "anisora_v1_diffusers.mp4", fps=16)

I’ll attach the diffusers output video in this comment. If you’re seeing a specific artifact you want addressed, let me know the exact behavior and I’ll dig deeper.
anisora_v1_diffusers.mp4

which transformers version you are using?

I didn't pin the transformers version in that comparison — I ran pip install transformers which installed the latest version compatible with diffusers 0.36.0. I don't have that environment anymore so I can't check the exact version. I can re-create the environment and report back with the exact version if needed.

did you keep the seed as the same for comparision with diffusers?

Good catch — I didn't set a fixed seed in the diffusers baseline script. The comparison was qualitative, showing that the same motion/shape characteristics appear in both implementations. I can re-run with a fixed seed in both (generator=torch.Generator("cuda").manual_seed(42)) if needed.

@vllm-omni-reviewer

🤖 VLLM-Omni PR Review

Code Review: Add Index-AniSora I2V Support

1. Overview

This PR adds support for Index-AniSora Image-to-Video models, supporting both the 5B (CogVideoX-based) and 14B (Wan2.1-based) variants. The implementation includes:

• Two new pipeline modules with hybrid loading for V2
• Registry and entrypoint integration
• Documentation and example updates

Overall Assessment: LGTM with suggestions - The implementation is well-structured and follows existing patterns, but has several issues that should be addressed before merging.

2. Code Quality

Positive Aspects

• Well-documented code with clear docstrings and comments
• Good logging throughout for debugging
• Clean separation between V1 and V2 architectures
• Proper type hints used consistently

Issues Found

Critical: Incorrect ValueError usage

vllm_omni/diffusion/models/anisora/pipeline_anisora_i2v_cogvideox.py:79-82

raise ValueError(
    """No image is provided. This model requires an image to run.""",
    """Please correctly set `"multi_modal_data": {"image": <an image object or file path>, …}`""",
)

This raises ValueError with two arguments, which creates a tuple exception message. Same issue at lines 85-88 and in pipeline_anisora_v2_i2v.py:77-84.

Fix:

raise ValueError(
    "No image is provided. This model requires an image to run. "
    "Please correctly set `multi_modal_data: {image: <an image object or file path>, …}`"
)

Potential Bug: Duplicate weight loading

vllm_omni/diffusion/models/anisora/pipeline_anisora_v2_i2v.py:248-252

def load_weights(self, weights: Iterable[tuple[str, torch.Tensor]]) -> set[str]:
    """Load weights using AutoWeightsLoader for vLLM integration."""
    loader = AutoWeightsLoader(self)
    return loader.load_weights(weights)

The V2 pipeline loads weights manually in __init__ (lines 196-243), but also provides load_weights. If vLLM calls load_weights after initialization, weights could be loaded twice or incorrectly.

Suggestion: Either remove load_weights for V2 or make __init__ not load weights and rely on vLLM's weight loading mechanism.

3. Architecture & Design

Positive Aspects

• Hybrid loading approach for V2 is well-documented and necessary for community weight compatibility
• Pre/post-process functions follow existing patterns in the codebase
• Key name conversion logic is comprehensive and clearly documented

Concerns

Fragile model detection logic

vllm_omni/entrypoints/omni_diffusion.py:71-77

if (
    class_name == "CogVideoXImageToVideoPipeline"
    and "anisora" in os.path.basename((od_config.model or "").rstrip("/")).lower()
):
    class_name = "AniSoraI2VCogVideoXPipeline"

This string matching could match unintended models (e.g., /models/anisora_experiment/other-model). Consider:

1. Adding a config file check for AniSora-specific markers
2. Using a more specific pattern match
3. Documenting the naming convention requirement

Forced download in V2 pipeline

vllm_omni/diffusion/models/anisora/pipeline_anisora_v2_i2v.py:196-200

if local_anisora:
    weight_path = model_path
else:
    weight_path = snapshot_download(model_path, local_files_only=False)

The local_files_only=False forces network access even when files might be cached. Should respect offline mode:

weight_path = snapshot_download(model_path, local_files_only=local_anisora)

Missing offline mode support for Wan base

vllm_omni/diffusion/models/anisora/pipeline_anisora_v2_i2v.py:167-175

The Wan2.1 base components are loaded with local_files_only=local_wan, but local_wan is determined by checking if the default path exists locally, which will almost always be False for the default HuggingFace ID.

4. Security & Safety

Input Validation

Missing path validation

vllm_omni/diffusion/models/anisora/pipeline_anisora_v2_i2v.py:201-206

safetensor_files = glob.glob(os.path.join(weight_path, "*.safetensors"))
if not safetensor_files:
    safetensor_files = glob.glob(os.path.join(weight_path, "**/*.safetensors"), recursive=True)

The glob patterns could potentially match unintended files. Consider:

1. Validating that files are within the expected directory
2. Filtering for specific expected file patterns

Silent failure for missing CLIP encoder

vllm_omni/diffusion/models/anisora/pipeline_anisora_v2_i2v.py:177-184

except Exception as e:
    logger.warning("CLIP image encoder not available: %s", e)
    self.image_processor = None
    self.image_encoder = None
    self.has_image_encoder = False

Catching broad Exception and continuing could mask real issues. The I2V pipeline may produce degraded results without CLIP conditioning. Consider:

1. Making CLIP encoder required for I2V models
2. Or raising a clear warning that output quality may be affected

5. Testing & Documentation

Documentation

• ✅ README updated with new models
• ✅ Usage examples provided
• ✅ Demo results with specific settings
• ✅ Technical notes explain hybrid loading approach

Missing

• ❌ No unit tests for the new pipelines
• ❌ No validation tests for key conversion logic
• ❌ No tests for model detection logic in entrypoint

Suggested Test Cases

1. Key conversion correctness (compare converted keys against expected mapping)
2. Model detection logic for various model name patterns
3. Error handling for missing images
4. Dimension validation for various input sizes

6. Specific Suggestions

pipeline_anisora_i2v_cogvideox.py

pipeline_anisora_v2_i2v.py

omni_diffusion.py

registry.py

7. Approval Status

LGTM with suggestions

The PR is well-structured and follows existing patterns in the codebase. The hybrid loading approach for V2 is necessary and well-implemented. However, the following should be addressed:

Required before merge:

1. Fix the ValueError multi-argument issue (affects error messages displayed to users)

Recommended:
2. Fix the local_files_only=False forced download in V2 pipeline
3. Add basic unit tests for key conversion logic
4. Document the model naming convention requirement for auto-detection

Optional improvements:
5. Consider making CLIP encoder required for V2 I2V
6. Add validation for the load_weights vs __init__ weight loading in V2

This review was generated automatically by the VLLM-Omni PR Reviewer Bot
using glm-5.

Thanks for the thorough review and the LGTM! I really appreciate you taking the time to go through everything.

The @vllm-omni-reviewer bot flagged a few additional items — most were false positives or already covered, but it did catch a real bug: our ValueError calls were passing two string arguments (creating a tuple message) instead of a single concatenated string. Just pushed a fix for that in 57e3eb0.

All feedback addressed — ready for merge whenever you're comfortable!

@wtomin PTAL

@wtomin

PR Update: Benchmarks, E2E Tests, SP Support & V2 TP Bug Report

What this PR adds

• V1 (5B, CogVideoX-based) I2V pipeline with Ulysses sequence parallelism (SP), tensor parallelism (TP), and FP8 quantization
• V2 (14B, Wan2.1-based) I2V pipeline with AniSora→diffusers weight key conversion and TP support
• Benchmark script (benchmark_anisora.py) with --warmup, --tp, --sp, --quantization flags
• E2E offline tests: single GPU, TP=2, SP=2, FP8, V2 TP=2
• E2E online serving tests: full job lifecycle (create → poll → download → delete)

Benchmark Results (2× H100 80GB)

• SP=2 gives ~34% speedup vs TP=1 (131.77s vs 200.75s)
• FP8 reduces VRAM by ~14% (76.86 → 65.95 GiB) with no latency impact

E2E Test Results

Offline inference — 5/5 passed (159s):

test_anisora_v1_offline_single_gpu  PASSED
test_anisora_v1_offline_tp2         PASSED
test_anisora_v2_offline_tp2         PASSED
test_anisora_v1_offline_sp2         PASSED
test_anisora_v1_offline_fp8         PASSED